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Abstract — Microgrid (MG) is a promising component for 
future smart grid (SG) deployment. The balance of supply 
and demand of electric energy is one of the most important 
requirements of MG management. In this paper, we present a 
novel framework for smart energy management based on the 
concept of quality-of-service in electricity (QoSE). Specifically, 
the resident electricity demand is classified into basic usage and 
quality usage. The basic usage is always guaranteed by the MG, 
while the quality usage is controlled based on the MG state. The 
microgrid control center (MGCC) aims to minimize the MG 
operation cost and maintain the outage probability of quality 
usage, i.e., QoSE, below a target value, by scheduling electricity 
among renewable energy resources, energy storage systems, and 
macrogrid. The problem is formulated as a constrained stochastic 
programming problem. The Lyapunov optimization technique is 
then applied to derive an adaptive electricity scheduling algo- 
rithm by introducing the QoSE virtual queues and energy storage 
virtual queues. The proposed algorithm is an online algorithm 
since it does not require any statistics and future knowledge of 
the electricity supply, demand and price processes. We derive 
several "hard" performance bounds for the proposed algorithm, 
and evaluate its performance with trace-driven simulations. The 
simulation results demonstrate the efficacy of the proposed 
electricity scheduling algorithm. 

Index Terms — Smart grid, Microgrids, distributed renewable 
energy resource, Lyapunov optimization, stability. 



I. Introduction 

Smart grid (SG) is a modem evolution of the utility elec- 
tricity delivery system. SG enhances the traditional power 
grid through computing, communications, networking, and 
control technologies throughout the processes of electricity 
generation, transmission, distribution and consumption. The 
two-way flow of electricity and real-time information is a 
characteristic feature of SG, which offers many technical 
benefits and flexibilities to both utility providers and con- 
sumers, for balancing supply and demand in a timely fashion 
and improving energy efficiency and grid stability. According 
to the US 2009 Recovery Act El, an SG will replace the 
traditional system and is expected to save consumer cost and 
reduce America's dependence on foreign oil. These goals are 
to be achieved by improving efficiency and spurring the use 
of renewable energy resources. 

Microgrid (MG) is a promising component for future SG 
deployment. Due to the increasing deployment of distributed 
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Fig. 1. Illustrate the microgrid architecture. 



renewable energy resources (DRERs), MG provides a local- 
ized cluster of renewable energy generation, storage, distri- 
bution and local demand, to achieve reliable and effective 
energy supply with simplified implementation of SG func- 
tionalities El, m. A typical MG architecture is illustrated 
in Fig. [TJ consisting of DRERs (such as wind turbines and 
solar photovoltaic cells), energy storage systems (ESS), a 
communication network (e.g., wireless or powerline commu- 
nications) for information delivery, an MG central controller 
(MGCC), and local residents. The MG has centralized control 
with the MGCC 141, which exchanges information with local 
residents, ESS's, and DRERs via the information network. 
There is a single common coupling point with the macrogrid. 
When disconnected, the MG works in the islanded mode and 
DRERs and ESS's provide electricity to local residents. When 
connected, the MG may purchase extra electricity from the 
macrogrid or sell excess energy back to the market fS]. 

The balance of electricity demand and supply is one of 
the most important requirements in MG management. Instead 
of matching supply to demand, smart energy management 
matches the demand to the available supply using direct load 
control or off-peak pricing to achieve more efficient capacity 
utilization (S). In this paper, we develop a novel control 
framework for MG energy management, exploiting the two- 
way flows of electricity and information. In particular, we 
consider two types of electricity usage: (i) a pre-agreed basic 
usage that is "hard"-guaranteed, such as basic living usage, 
and (ii) extra elastic quality usage exceeding the pre-agreed 
level for more comfortable life, such as excessive use of air 
conditioners or entertainment devices. In practice, residents 
may set their load priority and preference to obtain the two 
types of usage |6|. The basic usage should be always satisfied, 
while the quality usage is controlled by the MGCC according 
to the grid status, such as DRER generation, ESS storage 
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levels and utility prices. The MGCC may block some quality 
usage demand if necessary. This can be implemented by in- 
corporating smart meters, smart loads and appliances that can 
adjust and control their service level through communication 
flows f5l. To quantify residents' satisfaction level, we define 
the outage percentage of the quality usage as Quality of Service 
in Electricity (QoSE), which is specified in the service con- 
tracts [7 |. The MGCC adaptively schedules electricity to keep 
the QoSE below a target level, and accordingly dynamically 
balance the load demand to match the available supply. 

In this paper, we investigate the problem of smart energy 
scheduling by jointly considering renewable energy distri- 
bution, ESS management, residential demand management, 
and utility market participation, aiming to minimize the MG 
operation cost and guarantee the residents' QoSE. The MGCC 
may serve some quality usage with supplies from the DRERs, 
ESS's and macrogrid. On the other hand, the MG can also sell 
excessive electricity back to the macrogrid to compensate for 
the energy generation cost. The electricity generated from re- 
newable sources is generally random, due to complex weather 
conditions, while the electricity demand is also random due 
to the random consumer behavior, and so do the purchasing 
and selling prices on the utility market. It is challenging to 
model the random supply, demand, and price processes for 
MG management, and it may also be costly to have precise, 
real-time monitoring of the random processes. Therefore, a 
simple, low cost, and optimal electricity scheduling scheme 
that does not rely on any statistical information of the supply, 
demand, and price processes would be highly desirable. 

We tackle the MG electricity scheduling problem with a 
Lyapunov optimization approach, which is a useful technique 
to solve stochastic optimization and stability problems O . We 
first introduce two virtual queues: QoSE virtual queues and 
battery virtual queues to transform the QoSE control problem 
and battery management problem to queue stability problems. 
Second, we design an adaptive MG electricity scheduling 
policy based on the Lyapunov optimization method and prove 
several deterministic (or, "hard") performance bounds for the 
proposed algorithm. The algorithm can be implemented online 
because it only relies on the current system status, without 
needing any future knowledge of the energy demand, supply 
and price processes. The proposed algorithm also converges 
exponentially due to the nice property of Lyapunov stability 
design |9|. The algorithm is evaluated with trace-driven sim- 
ulations and is shown to achieve significant efficiency on MG 
operation cost while guaranteeing the residents' QoSE. 

The remainder of this paper is organized as follows. We 
present the system model and problem formulation in Sec- 
tion ini An adaptive MG electricity scheduling algorithm is 
designed and analyzed in Section |llll Simulation results are 
presented and discussed in Section Hv] We discuss related 
work in Section (V] Section |Vl] concludes the paper. 

II. System Model and Problem Formulation 

A. System Model 

1) Overview: We consider the electricity supply and con- 
sumption in an MG as shown in Fig. [TJ We assume that the 



TABLE I 
Notation 

Symbol Descnption 

IV total number ot residents 

K total number of batteries 

T total number of slots 

Ek (t) energy level for battery k at time slot t 

Rk (t) recharging energy for battery k at time slot t 

Dk(t) discharging energy for battery k at time slot t 

^rnax maximum battery energy level for battery k 

^m^r^ minimum battery energy level for battery k 

j^9hax maximum supported recharging energy for batter /c in a slot 

jjrnax maximum supported discharging energy for battery in a slot 

An average quality usage arrival rate for resident n 

pn average outage rate of quality usage for resident n in MG 

6n target QoSE for resident n in MG 

an(t) quality usage of residents n in time slot t 

(j^rnax maximum quality usage of resident n in a single slot 

a^(t) basic electricity usage of resident n in time slot t 

P(t) available electricity from DRERs to supply quality usage in 
time slot t 

U(t) electricity generated from DRERs in time slot t 

Q(t) electricity purchased from macrogrid in time slot t 

S(t) electricity sold on the market in time slot t 

Pn(t) electricity to the resident n 

C(t) purchasing price on the utility market in time slot t 

W(t) selling price ob the utility market in time slot t 

In(t) indicator function for outage events of quality usage of 

resident n in time slot t 

Cmin minimum purchasing price of utility from macrogrid 

Cmax maximum purchasing price of utility from macrogrid 

^min minimum selling price of utility to macrogrid 

Wmax maximum selling price of utility to macrogrid 

Xk(t) battery virtual queue for the battery k 

Zn(t) QoSE virtual queue for the resident n 

B(t) states of the virtual queues X]^(t) and Zn{t) 

L(-) Lyapunov function 

A(t) Lyapunov one step drift 

A(t) proposed scheduling policy including S{t)^ Rk{t), 

Dk(t) and pn{t) 

y* optimal objective value of problem (|9j 

A(t) relaxed scheduling policy for problem [25l 

y optimal objective value of problem ( I25t 



MG is properly designed such that a portion of the electricity 
demand related to basic living usage (e.g., lighting) from 
the residents, termed basic usage, can be guaranteed by the 
minimum capacity of the MG. There are randomness in both 
electricity supply (e.g., weather change) and demand (e.g., en- 
tertainment usage in weekends). To cope with the randomness, 
the MG works in the grid-connected mode and is equipped 
with ESS's, such as electrochemical battery, superconducting 
magnetic energy storage, flywheel energy storage, etc. The 
ESS's store excess electricity for future use. 

The MGCC collects information about the resident de- 
mands, DRER supplies, and ESS levels through the informa- 
tion network. When a resident demand exceeds the pre-agreed 
level, a quality usage request will be triggered and transmitted 
to the MGCC. The MGCC will then decide the amount of 
quality usage to be satisfied with energy from the DRERS, 
the ESS's, or by purchasing electricity from the macrogrid. 
The MGCC may also decline some quality usage requests. 
The excess energy can be stored at the ESS's or sold back to 
the macrogrid for compensating the cost of MG operation. 

Without loss of generality, we consider a time-slotted sys- 
tem. The time slot duration is determined by the timescale of 
the demand and supply processes. 
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Fig. 2. The system model considered in this paper. 



2) Energy Storage System Model: The system model is 
shown in Fig. [2l Consider a battery farm with K independent 
battery cells, which can be recharged and discharged. We 
assume that the batteries are not leaky and do not consider the 
power loss in recharging and discharging, since the amount is 
usually small. It is easy to relax this assumption by apply- 
ing a constant percentage on the recharging and discharging 
processes. For brevity, we also ignore the aging effect of the 
battery and the maintenance cost, since the cost on the utility 
market dominates the operation cost of MGs. 

Let Ek{t) denote the energy level of the the kth battery in 
time slot t. The capacity of the battery is bounded as 



<Ek{t) KE^j^^'^^Mk^t 



(1) 



where 



> is the maximum capacity, and E]^^^ > is 



the minimum energy level required for battery which may 
be set by the battery deep discharge protection settings. The 
dynamics over time of Ek (t) can be described as 



Ek{t^l)= Ek(t) - Dk(t) + Rk(t),\/k, t, 



(2) 



where Rk{t) and Dk{t) are the recharging and discharging 
energy for battery k in time slot t, respectively. The charging 
and discharging energy in each time slot are bounded as 



< Rk{t) < \Jk,t 
{) < Dk(t) < D^^^^ yk,t. 



(3) 



In each time slot t, Rk{t) and Dk{t) are determined such that 
([B is satisfied in the next time slot. 

Usually the recharging and discharging operations cannot 
be performed simultaneously, which leads to 



Rk{t) > 0^ Dk{t) =0, W k,t 
Dk(t) >0^ Rklt) =0, \f k,t. 



(4) 



3) Energy Supply and Demand Model: Consider N resi- 
dents in the MG; each generates basic and quality electricity 
usage requests, and each can tolerate a prescribed outage 
probability Sn for the requested quality usage part. The MGCC 
adaptively serves quality usage requests at different levels to 
maintain the QoSE as well as the stability of the grid. The 
service of quality usage can be different for different residents, 
depending on individual service agreements. 



Let An be the average quality usage arrival rate, and Sn a 
prescribed outage tolerance (i.e., a percentage) for user n. The 
average outage rate for the quality usage, pn, should satisfy 



Pn ^ Sn ' Arj 



(5) 



At each time t, the quality usage request from resident n is 
an{t) G [0,^5^'^^] units, which is an i.i.d random variable 
with a general distribution and mean A^. The average rate 
is An = limt^oo(l/^) Xlt=o ^^('^) according to the Law of 
Large Numbers. 

The DRERs in the MG generate U{t) units of electricity 
in time slot t. U (t) can offer enough capacity to support the 
pre-agreed basic usage in the MG, which is guaranteed by 
islanded mode MG planning. The electricity is transmitted 
over power transmission lines. Without loss of generality, we 
assume the power transmission lines are not subject to outages 
and the transmission loss is negligible. Let a^{t) be the pre- 
agreed basic usage for resident n in time slot t, which can be 
fully satisfied by U{t), i.e., Y^^^^ a^^{t) < U{t), for all t In 
addition, some quality usage request an{t) may be satisfied 
if P{t) = U{t) - En=i ^n{t) > 0- Let pn{t) be the energy 
allocated for the quality usage of resident n. We have 



0<Pn{t)<an{t). 



(6) 



We define a function In{t) > to indicate the amount 
of quality usage outage for resident n, as In{t) = Q^n(^) — 
Pn{t). Then the average outage rate can be evaluated as pn = 

lim,^oo(lA) Er 

The MGCC may purchase additional energy from the 
macrogrid or sell some excess energy back to the macrogrid. 
Let Q{t) e [O^Qmax] denote the energy purchased from the 
macrogrid and S{t) G [0, Smax] the energy sold on the market 
in time slot t, where Qmax and Smax are determined by the 
capacity of the transformers and power transmission lines. 
Since it is not reasonable to purchase and sell energy on the 
market at the same time, we have the following constraints 



(7) 



r Q{t) >o^ s{t) = 0, yt 

\ S{t) >O^Q(t) =0, Vt. 
To balance the supply and demand in the MG, we have 

K K N 

Pit) + Q(t) + W) - Sit) -Y.Rk{t) = Y^ Pn{t)y t. 



(8) 

4) Utility Market Pricing Model: The price for purchasing 
electricity from the macrogrid in time slot t is C{t) per 
unit. The purchasing price depends on the utility market 
state, such as peak/off time of the day. We assume finite 
C{t) G [CminjCmax], which is announccd by the utility 
market at the beginning of each time slot and remains constant 
during the slot period [10|. Unlike prior work [|TOl . we do not 
require any statistic information of the C{t) process, except 
that it is independent to the amount of energy to be purchased 
in that time slot. 

If the MGCC determines to sell electric energy on the 
utility market, the selling price from the market broker is 
denoted by W{t) G [Wniin,^max] in time slot t, which is 
also a stochastic process with a general distribution. We also 
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assume W{t) is known at the beginning of each time slot and 
independent to the amount of energy to be sold on the market. 

We assume Cmax ^ ^^max? Cmin ^ ^^min 3-Ild C{t) > ^if) 

for all t. That is, the MG cannot make profit by greedily 
purchasing energy from the market and then sell it back to 
the market at a higher price simultaneously. 

B. Problem Formulation 

Given the above models, a control policy K(t) = 
{(5(t), S{t)^ Rk{t)^ Dk{t)^Pn{t)} is designed to minimize the 
operation cost of the MG and guarantee the QoSE of the 
residents. We formulate the electricity scheduling problem as 

1 '"^ 

minimize: \im -^^ E{Q(r)C(r) - SMWir)} (9) 

t^oo t ^ — ^ 



=0 



s.t. O, ®, (S, ®, (©, dU), dsl) 

battery queue stability constraints. 

Problem © is a stochastic programming problem, where 
the utility prices, generation of DRERs, and consumption of 
residents are all random. The solution also depends on the 
evolution of battery states. It is challenging since the supply, 
demand, and price are all general processes. 

1) Virtual Queues: We first adopt a battery virtual queue 
Xk{t) that tracks the charge level of each battery k\ 

Xk{t) = Ek{t) - - - VCmax. V k,t, (10) 

Cm ax in J 



where < F < 



Cm a X ^^rr 

is a constant for the trade-off between system performance 
and ensuring the battery constraints. This constant Vmax 
is carefully selected to ensure the evolution of the battery 
levels always satisfy the battery constraints Q, which will be 
examined in Section IIII-CI The virtual queue can be deemed 
as a shifted version of the battery dynamics in ^ as 

Xk(t^l) = Xk(t)-Dk(t)^Rk(t), yk,t (11) 

These queues are "virtual" because they are maintained by the 
MGCC control algorithm. Unlike an actual queue, the virtual 
queue backlog Xk{t) may take negative values. 

We next introduce a conceptual QoSE virtual queue Zn{t), 
whose dynamics are governed by the system equation as 

Zn{t^l) = [Zn{t)-Sn-an{t)]^ ^In{t), V n, t. (12) 

where [x]~^ = max{0,x}. 

Theorem 1. If an MGCC control policy stabilizes the QoSE 
virtual queue Zn{t), the outage quality usage of resident n 
will be stabilized at the average QoSE rate pn < Sn - Xn- 

Proof: According to the system equation ([T2l) . we have 

Zn 

(13) 

Zr^it) > Znit -l)-5n- an{t - 1) + In{t - 1). 

Summing up the inequalities in ([13]) , we have 

t-i t-i 
Zn{t) > Zn{0) - Sn ' ^^^(r) + ^/n(r). (14) 



Dividing both sides by t and letting t go to infinity, we have 

t-i t-i 



t^oo t t^oo t 



mmimize: 



s.t. 



r=0 r=0 

Note that Zn{0) is finite. If Zn{t) is rate stable by 
a control policy In{t), it is finite for all t. We have 
lim^^oo ^^'^^^"^^'^^^ = 0, which yields pn < ' due 
to the definitions of and In{t). ■ 
2) Problem Reformulation: With Theorem [T] we can trans- 
form the original problem ^ into a queue stability problem 
with respect to the QoSE virtual queue and the battery 
virtual queues, which leads to a system stability design from 
the control theoretic point of view. We have a reformulated 
stochastic programming problem as follows. 

1 '"^ 

lim - Y,m{r)C{r)-S{r)W{r)} (15) 

©, (S, (©, 0, © 

Battery and QoSE virtual queue stability 
constraints. 

Theorem \T\ indicates that QoSE provisioning is equivalent to 
stabilizing the QoSE virtual queue Zn{t), while stabilizing 
the virtual queues (fTTt ensures that the battery constraints ([B 
are satisfied. We then apply Lyapunov optimization to develop 
an adaptive electricity scheduling policy for problem (fTSl l, in 
which the policy greedily minimize the Lyapunov drift in every 
slot t to push the system toward stability. 

C. Lyapunov Optimization 

We define the Lyapunov function for system state 6(t) = 
[X(t), Z{t)]^ with dimension {N-{-K) x 1 as follows, in which 
X{t) = [Xi{t) ■ ■ ■ XK{t)f and Z{t) = [Z,{t) ■ ■ ■ ZN{t)f. 

K N 

Lim) = - ^ [X,{t)f + - ^ [Z^itf , (16) 



n=l 



which is positive definite, since L{Q{t)) > when 6(t) 7^ 
and L{Q{t)) = ^ Q{t) = 0. We then define the conditional 
one slot Lyapunov drift as 

A{e{t)) = E{L{e{t + 1)) - L{emm}. (ii) 

With the drift defined as in (fTTl) , it can be shown that 
A(e(t)) = iE|f][(Xfe(t+l))2-(Xfe(t))2|Xfc(t)] + 

N ^ 



J2[iZnit + i)f-iz„it)f\z„it)]\ 

n=l J 

N 

< B^^E{Zn{t){l-Sn)an{t)\Zn{t)}^ 

n=l 

K 

Y,¥.{Xk{t){Rk{t) - Dum^km - 

N 

Y,H{Zn{t) ^ an{t))Pn{t)\Zn{t)}, (18) 



r=0 



r=0 
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where B = i Ef=i(max{D--, iir^)' + ^E„=i(2 + 
^n)(^n^^^)^ is a constant. The derivation of ([TS]) is given in 
Appendix (A] 

To minimize the operation cost of the MG, we adopt 
the drift-plus-penalty method ifTTIl . Specifically, we select the 
control policy A{t) = {Q{t), S{t), Rk{t), Dk{t),pn{t)} to 
minimize the bound on the drift-plus-penalty as: 

A(6(t)) + VE{Q{t)C{t) - S{t)W{t)\G{t)} 
< right-hand- side of (fTSl) + 

VE{Q{t)C{t) - S{t)W{t)\e{t)}, (19) 

where < V < Vmax is defined in Section III-Bll for the 
trade-off between stability performance and operation cost 
minimization. Given the current virtual queue states 
and Zn(t\ market prices S(i) and W(t\ available DRERs 
energy and the resident quality usage request an(t), the 
optimal policy is the solution to the following problem. 

N 

minimize: 

n=l 

V\Q{t)C{t) - S{t)W{f)\ + 

K 

Yy^k{t){Rkit)-Dm\- 

N 

Y^^^n(t)^OLn(t))Pn(t)\ (20) 
n=l 

S.t. ©, ©, ©, ©, dU) . 

Since the control policy A(t) is only applied to the last three 
terms of (l2Ql) . we can further simplify problem (l2Ql as 

K 

minimize: V\Q(t)C(t) - S(t)W(t)\ + ^ \X^(i){R^(t)- 

k=l 

N 

Dkim - ^[(^nW + an{t))pn{t)] (21) 

S.t. ©, ©, ©, ©, (E) , 

which can be solved based on observations of the current 
system state {Xk{t), Zn{t),C{t),W{t), P{t),an{t)}. 

III. Optimal Electricity Scheduling 

A. Properties of Optimal Scheduling 

With the Lyapunov penalty-and-drift method, we transform 
problem (fTSl) to problem (1211 to be solved for each time slot. 
The solution only depends on the current system state; there 
is no need for the statistics of the supply, demand and price 
processes and no need for any future information. The solution 
algorithm to this problem is thus an online algorithm. We have 
the following properties for the optimal scheduling. 

Lemma 1. The optimal solution to problem d271) has the 

following properties: 

1) IfQ{t) > 0, we have S{t) = 0, 

a) If Xk{t) > —VC{t), the optimal solution always 
selects Rk{t) = 0; if Xk{t) < -VC{t), the 
optimal solution always selects Dkit) = 0. 



b) If Zn{t) > VC{t) — an{t)y the optimal solution 
always selects Pn{t) > (1 — Sn)(^n{t); if Zn{t) < 
VC{t) — an{t), the optimal solution always selects 

Pn{t)=0. 

2) When Q{t) = 0, we have S{t) > 0, 

a) If X kit) > —VW{t), the optimal solution always 
selects Rk{t) = 0; if Xk(t) < -VW{t), the 
optimal solution always selects Dkit) = 0. 

b) If Zn{t) > VW{t) — an{t)y the optimal solution 
always selects Pn{t) > (1 — Sn)(^n{t); if Zn{t) < 
VW{t) — an{t), the optimal solution always selects 

Pn{t)=0. 

The proof of Lemma [T] is given in Appendix IbI 

Lemma 2. The optimal solution to the battery management 
problem has the following properties: 

1) If Xk{t) > —VWminy the optimal solution always 
selects Rk{t) = 0. 

2) If Xk{t) < —VCmaxy the optimal solution always 
selects Dkit) = 0. 

The proof of Lemma [2] is given in Appendix O 

Lemma 3. The optimal solution to the QoSE provisioning 
problem has the following properties: 

1) If Znit) > VCmax^ the Optimal solution always selects 

Pn{t) > {1-Sn)an{t). 

2) If Zn{t) < VWmin — o^max^ the Optimal solution always 
selects Pnit) = 0. 

The proof directly follows Lemma [T] and is similar to the 
proof of Lemma [21 We omit the details for brevity. 

Lemma [T] provides useful insights for simplifying the al- 
gorithm design, which will be discussed in Section IIILBI 
The intuition behind these lemmas is two-fold. On the ESS 
management side, if either the purchasing price C{t) or the 
selling price W{t) is low, the MG prefers to recharge the 
ESS's to store excess electricity for future use. On the other 
hand, if either Cit) or W{t) is high, the MG is more likely 
to discharge the ESS's to reduce the amount of energy to 
purchase or sell more stored energy back to the macrogrid. 
On the QoSE provisioning side, if either C{t) or W{t) is high 
and the quality usage an{t) is low, the MG is apt to decline 
the quality usage for lower operation cost. On the other hand, 
if either C{t) or W{t) is low and an{t) is high, the quality 
usage are more likely to be granted by purchasing more energy 
or limiting the sell of energy. 



B. MG Optimal Scheduling Algorithm 

In this section, we present the MG control policy Kit) 
to solve problem (|2T1) . Given the current virtual queue state 
{Xk{t)^ Zn{t)}, market prices C{t) and W{t), quality usage 
an{t) and available energy P{t) from the DRERS for serving 
quality usage, problem (|2T1) can be decomposed into the 
following two linear programming (LP) sub-problems (since 



5 



Algorithm 1: Adaptive Electricity Scheduling Algorithm 

1 MGCC initializes the QoSE target to 5n and the virtual 
queues backlogs Zn{t) and Xk{t), for all n and k ; 

2 while TRUE do 
Residents send usage request (with basic and quality 
usage) to MGCC via the information network ; 
MGCC solves LPs ^ and ([23]) ; 
MGCC selects the optimal solution A{t) comparing 
the solutions to and ([23]) ; 
MGCC updates the virtual queues Xk{t) and Zn{t) 
according to ([TT1) and ([T2l) . for all n and k ; 

end 



one of S{t) and Q{t) must be zero, see (O). 

K 

minimize: VQ{t)C{t) + ^[Xfc(t)(i?fc(t) - Dkit))] 



AT 



^((Zn(t)+an(t)KW) 
s.tS{t) = OM. ©, ([6]), ([8]). 



(22) 



K 



minimize: -F^(t)I^(t) + ^[Xfe(t)(i?fe(t) - D^m 



AT 



^((Zn(t) + anWKW) 
s.t. g(t) = o,([3]), (a, ([g, 



(23) 



In sub-problem ([22]), we set Rk{t) = if Xk{t) > -VC{t), 
and D/c(t) = if Xk{t) < -VC{t) according to Lemma [T] 
Also, if Zn{t) < VC{t) - an{t), we set pn{t) = 0; 
otherwise, we reset constraint ([6J to a smaller search space of 
{1—Sn)an{t) < Pn{t) < (^n{t). We take a similar approach for 
solving sub-problem ([23]) by replacing C{t) with W{t). Then 
we compare the objective values of the two sub-problems and 
select the more competitive solution as the MG control policy. 
The complete algorithm is presented in Algorithm [T] 

C. Performance Analysis 

The proposed scheduling algorithm dynamically balances 
cost minimization and QoSE provisioning. It only requires 
current system state information (i.e., as an online algorithm) 
and requires no statistic information about the random supply, 
demand, and price processes. The algorithm is also robust to 
non-i.i.d. and non-ergodic behaviors of the processes [[TTI . 



Theorem 2. The constraint on the ESS battery level Ekif), 
< Ek it) < EJ^^^, is always satisfied for all k and t. 

Proof: From the battery virtual queue definition ([TQ] |. the 
< Ek{t) < is equivalent to 



constraint 



-va 



r\max ^ v" ( 4-\ ^ T7''m 



-va 



T~\max rprmn 

max J^k 



We assume all the batteries satisfy the battery capacity con- 
straint at the initial time t 



0, i.e., E]^'"" < Ek{0) < E- 



max 
k 



for all k. Supposing the inequalities hold true for time t, we 
then show the inequalities still hold true for time t + 1. 
First, we show Xk{t^l) < E]^^"" -VCmax- 

If -VWrmn < Xk(t) < E^^^ - VCm. 

then with Xk{t) > -VWmin Rk{t) = from Lemma [2] 
we have Xk{t + 1) = Xk{t) - Dk(t) < Xk{t) < E^^^ - 
VCmax - D^^^ - E^'^. If Xk{t) < -VWrmn. then the 



Dmax rpmin 
-^k -^k • 
T^max rpmin 
^k - ^/c ' 



largest value is X/e(t + 1) = 
< F < Vmax, we have 



-VWrr 



^raax ^ For any 



Tpmax \rr^ r^max rrimtn 

^k - ^ ^max - J^k - ^k 



rpmax rpmzn iDmax r^max 

^k - ^k - ^k - ^k I r 

O77 



-D 



max -^mzn ^ -^mzn _|_ j^max ^ Xf^(j; \ 1) 



It follows that Xk{t + 1)< E]^"^ - VCmax - D^"'' - i^r'"- 
Next, we show Xk{t + 1)> -VCmax - -Or""^- Assuming 
-VCmax- D^''^ < Xk{t) < -VCmax, then fromLemmaEl 
we have Xk{t) < -VCmax => Dk{t) = 0. It follows that 



Xk{t + 1) = Xk{t) + Rk{t) > Xk{t) > -VCmax - D, 

If Xk{t) > —VCmax, following (fTOl . we have 



max 
k 



Xk{t+l) 



Xk{t) - Dk{t) + Rk{t) > Xu(t)-D^ 



> 



-va 



T~\max 

max J-^k 



.Jjraax^ ThuS the 



Therefore, we have Xk{t^l) > —VCm 
inequalities also hold true for time t + 1. 

It follows that E]^'"" < Ek(t) < E^""^ is satisfied under 
the optimal scheduling algorithm for slU k, t. ■ 



Theorem 3. The worst-case backlogs of the QoSE virtual 
queue for each resident n is bounded by Zn{t) < Z^^^ = 
y Cmax^ y for all n, t Moreover, the worst-case average 
amount of outage of quality usage for resident n in a period 
T is upper bounded by Z^^^ + Tc^nQ^n 

Proof: (i) We first prove the upper bound Z^^^. Initially, 



we have Zn(0) = < VCn 



(^max ^ Assume that in time 



slot t the backlog of the QoSE virtual queue of resident n 



satisfies Zn{t) < Z;^^^ = VCrr 



^ We then check 



the backlog at time t-\-l and show the bound still holds true. 

If Zn{t) > VCmax ^ following Lemma [3] the optimal 
scheduling for the quality usage of resident n satisfies Pn{t) > 
{1 — Sn)an{t)- From the virtual queue dynamics ([T2l) , we have 

Zn 

liZn{t) > Snan{t), WC haVC Zn{t^l) < Zn{t) < VCmax ^ 

a;^^^; otherwise, it follows that Zn{t + 1) < 5n(^n{t) < 

^ ^max ^ '-^n 

If Znit) < VCmax, We haVC Znit + I) < [Znit) - 

(5„a„(t)]++a;^"^. \fZn{t) > Sna„it), we have Z„(t+1) < 
Z„{t) - Snanit) + a^''^ < VCmax + a^''^ ; otherwise, we 
have Znit+1)< a™"^ < VC„ ^ -"'""^ 



max ~T 
max 



vc^ 



The 



Thus we have Zn(t ^ 1) < Z, 
proof of the QoSE virtual queue backlog bound is completed. 

(ii) Consider an interval [^1,^2] with length of T. Sum- 
ming ([T2]) from ti to ^2, we have Zn{t2 + 1) > Zn{ti) — 
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P„(t)] - TSna'^''^. It follows that 

r=ti 

■ 

Theorem 4. The average MG operation cost under the 
adaptive electricity scheduling algorithm in Algorithn\l\ y, 
is bounded as ^ y ^ -\- /V, where y^ is optimal 
operating cost and = B ^ Y.n^i ^T""^^ - ^n)OLT'' - 

Proof: From Theorem |2l the battery capacity constraints 
is met in each time slot with the adaptive control policy. Take 
expectation on ^ and sum it over the period [0,t — 1]: 

Since < Ek{t) < we divide both sides by t and 

let t go to infinity, to obtain 

t-i t-i 
lim - yE{Rk{T)} = lim - V E{I)fc(r)}, V k. (24) 

r=0 T=0 

Consider the the following relaxed version of problem (|9). 

1 '"^ 

minimize: lim - V E{Q(r)C(r) - 5'(r)W(r)} (25) 

t^oo t ^ — ^ 
r=0 

s.t. ©, ©, ©, ®, ©, ®, and (El. 

Since the constraints in problem (|25l ) are relaxed from that in 
problem (O, the optimal solution to problem ^ is also feasi- 
ble for problem (|251 ). The solution of (l25l ) does not depend on 
battery energy levels. Let the optimal solution for problem (1251) 
be A(t) = {(5(t),^(t),i^fe(4^feW.Pn(t)} and the corre- 
sponding object value is y < y*. According to the properties 
of optimality of stationary and randomized policies [[T2l . the 
optimal solution A{t) satisfies E{Rk{t) — Dk{t)} = and 
y = E{QiT)CiT) - SiT)WiT)}. 

We substitute solution A{t) into the right-hand- side of the 
drift- and-penalty ([T9l ). Since our proposed policy minimizes 
the right-hand- side of ([T9l ), we have 

A(e(t)) + vE{Q{t)c{t) - s{t)w{t)\e{t)} 

N 

< B^yE{Zn{t){l-Sn)an{t)\Zn{t)}^ 

n=l 

K 

Y,Xk{t)E{Rk{t)-bk{t)\Xk{t)} - 

k=l 
N 

Y,(Zn{t) + an{t))E{pn{t)\Zn{t)} + 
n=l 

VE{Q{t)C{t) - S(t)W(t)\6(t)} 

N 

< B^Y.^n""{'^-^n)cxT" •y\ 

n=l 

The second inequality is due to E{R}^{t) — £^/c(t)} = 0, < 
Zn(t) < Z^^^, an{t) > 0, pn{t) > 0, and y < y\ Taking 



expectation and sum up from to T — 1, we obtain 

T-l 

V¥.{Q{t)C{t) - S{t)W{t)} 

< T-B^ ^T-V -y^ - E{L(e(T))} + E{L((9(0))} 

< T • 5* + T • V • ^* + E{L(6(0))}. 

The second inequality is due to the nonnegative property of 
Lyapunove functions. Divide both sides by F • T and let T go 
to infinity. Since the initial system state 9(0) is finite, we have 
limT^oo ^ EL"o' VE{Q{t)C{t) - S{t)W{t)} < ^* + ^. ■ 
It is worth noting that the choice of V controls the optimal- 
ity of the proposed algorithm. Specifically, a larger V leads to a 
tighter optimality gap. However, from the proof of Theorem |2l 

V is limited by Vmax, which ensures the feasibility of the 
battery constraints. This is actually a similar phenomenon to 
the so-called performance-congestion trade-off |[T2l . Through 
the definition of Vmax (see Section Ill-Bll) . it can be seen that 
if we invest more on the individual storage components for 
a larger ESS capacity, the proposed algorithm can achieve a 
better performance (i.e., a smaller optimality gap). 

It is also worth noting that all the performance bounds 
of the proposed algorithm are deterministic, which provide 
"hard" guarantees for the performance of the proposed adap- 
tive scheduling policy in every time slot. Unlike probabilistic 
approaches, the proposed method provides useful guidelines 
for the MG design, while guaranteeing the MG operation cost, 
grid stability, and the usage quality of residents. 

IV. Simulation Study 

We demonstrate the performance of the proposed adaptive 
MG electricity scheduling algorithm through extensive simu- 
lations. We simulated an MG with 500 residents, where the 
electricity from DRERs is supplied by a wind turbine plant. 
We use the renewable energy supply data from the Western 
Wind Resources Dataset published by the National Renewable 
Energy Laboratory |[T3]. The ESS's consists of 100 PHEV Li- 
ion battery packs, each of which has a maximum capacity of 
16 kWh and the minimum energy level is 0. The battery can 
be fully charged or discharged within 2 hours lfT4l . 

The residents' pre-agreed power demand is uniformly dis- 
tributed in [2 kW, 25 kW], and the quality usage power is 
uniformly distributed in [0, 10 kW]. The MG works in the 
grid-connected mode and may purchase/sell electricity from/to 
the macrogrid. The utility prices in the macrogrid are obtained 
from ilSl and are time- varying. We assume the sell price 
by the broker is random and below the purchasing price in 
each time slot. The time slot duration is 15 minutes. The 
MGCC serves a certain level of quality usage according to 
the adaptive electricity scheduling policy. The QoSE target is 
set to Sn = 0.07 for all residents. The control parameter is 

V = Vmaxr unless otherwise specified. 

A. Algorithm Performance 

We first investigate the average QoSEs and total MG op- 
eration cost with default settings for a five-day period. We 
use MATLAB LP solver for solving the sub-problems (l22l) 
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and ([23]) . For better illustration, we only show the QoSEs 
of three randomly chosen users in Fig. [3] It can be seen 
that all the average QoSEs converge to the neighborhood of 
0.08 within 200 slots, which is close to the MG requested 
criteria 5n = 0.07. In fact the proposed scheme converges 
exponentially, due to the inherent exponential convergence 
property in Lyapunov stability based design Q. 

We also plot the MG operation traces from this simulation in 
Fig. [6l The energy for serving quality usage from the DEREs 
are plotted in Fig.^A). It can be seen that the DRERs generate 
excessive electricity from slot 150 to 200, which is more 
than enough for the residents. Thus, the MGCC sells more 
electricity back to the macrogrid and obtains significant cost 
compensation accordingly. In Fig. OB), we plot the traces of 
electricity trading, where the positive values are the purchased 
electricity (marked as brown bars), and the negative values 
represent the sold electricity (marked as dark blue bars). The 
MG operation costs are plotted in Fig. OC). The curve rises 
when the MG purchases electricity and falls when the MG 
sells electricity. From slot 150 to 200, the operation cost drops 
significantly due to profits of selling excess electricity from 
the DEREs. The operation cost is $418.10 by the end of the 
period, which means the net spending of the MG is $418.10 
on the utility market. 

We then examine the energy levels of the batteries in Fig. S] 
We only plot the levels of three batteries in the first 50 
time slots for clarity. The proposed control policy charges 
and discharges the batteries in the range of to 16 kWh, 
which falls strictly within the battery capacity limit. It can be 
seen that the amount of energy for charging or discharging 
in one slot is limited by 2 kWh in the figure, due to the 
short time slots comparing to the 2 -hour fully charge/discharge 
periods. For longer time slot durations and batteries with faster 
charge/discharge speeds, the variation of the energy level in 
Fig. |4] could be higher. However, Theorem |2] indicates that 
the feasibility of the battery management constraint is always 
ensured, if the control parameter V satisfies < F < Vmax- 

We next evaluate the performance of the proposed adaptive 
control algorithm under different values of control parameter 
V. For different values V = {Vmax, Vmax f"^, Vmax /^}, the 
QoSEs are stabilized at 0.081, 0.061, and 0.055, and the total 
operation cost are $418.10, $625.69, and $717.75, respec- 
tively. We find the QoSE decreases from 0.081 to 0.055, while 
the total operation cost is increased from $418.10 to $717.75, 
as Vmax is decreased. This demonstrates the performance- 
congestion trade-off as in Theorem 51 a larger V leads to a 
smaller objective value (i.e., the operating cost), but the system 
is also penalized by a larger virtual queue backlog, which 
corresponds to a higher QoSE. On the contrary, a smaller 
V favors the resident quality usage, but increases the total 
operation cost. In practice, we can select a proper value for 
this parameter based on the MG design specifications. 

It would be interesting to examine the case where the 
residents require different QoSEs. We assume 5 residents with 
a service contract for lower QoSEs. We plot the average QoSEs 
of three residents with V = Vmax I in Fig- El Resident 1 
prefers an outage probability b\ =0.02, while residents 2 and 
3 require an outage probability (^2 = (^3 = 0.07. It can be seen 



in Fig.[5lthat resident I's QoSE converges to 0.015, while the 
other two residents' QoSEs remains around 0.063. 

B. Comparison with a Benchmark 

We compare the performance of the proposed scheme with a 
heuristic MG electricity control policy (MECP), which serves 
as a benchmark. In MECP, the MGCC blocks quality usage 
requests simply by tossing a coin with the target probability. 
We use = 0.03 in the following simulations. If there is 
sufficient electricity from the DRERs, all the quality usage 
requests will be granted and the excess energy will be stored 
in the ESS 's. If there is still any surplus energy, the MGCC will 
sell it to the macrogrid. If there is insufficient electricity from 
the DRERs, the ESS's will be discharged to serve the quality 
usage requests. The MGCC will purchase electricity from the 
macrogrid if even more electricity is required. Finally, with 
a predefined probability, e.g., 0.5 in the following simulation, 
the MG purchases as much energy as possible to charge the 
ESS's. 

We run 100 simulations with different random seeds for a 
seven-day period. We assume in the first five days the resident 
behavior is the same as previous default settings. In the last 
two days, we assume the residents are apt to request more 
electricity (e.g., more activities in weekends) We assume in 
the last two days the resident pre-agreed basic usage power 
demand is uniformly distributed from 5 kW to 35 kW. The 
quality usage power is uniformly distributed from to 20 kW. 

We find that the proposed algorithm earns $947.27 from the 
utility market (with 95% confidence interval [950.65, 943.89]). 
The profit mainly comes from the abundant DRER gen- 
eration in the last two days, as shown in Fig. [71 MECP 
only earns $379.74 from the market (with 95% confidence 
interval [387.96, 371.52]), which is 60% lower than that of the 
proposed control policy. We also find that the QoSEs under the 
proposed control policy remains about 0.025, which is lower 
than the criteria (5^ = 0.03. This is because there are a sudden 
price jump from $27/MWh to $356/MWh in the afternoon of 
the last day. This sharp increment increases Cmax eight times 
and decreases the value of Vmax- Due to the performance- 
congestion trade-off, the QoSEs become smaller (lower than 
MECP's 0.03 level). 

V. Related Work 

SG is regarded as the next generation power grid with two- 
way flows of electricity and information. Several comprehen- 
sive reviews of SG technologies can be found in (Si, O. 
Recently, SG research is attracting considerable interest from 
the networking and communications communities IIT6I - II2II . 
For example, the design of wireless communication systems 
in SG is studied in ifTTl . The authors of ifTSll . lfT9l explore 
the important wireless communication security issues in smart 
grid. The energy management and power flow control in the 
grid is investigated in |[T6l to reach system-wide reliability 
under uncertainties. The frequency oscillation in power net- 
works is studied in ll20l by epidemic propagation and a social 
network based approach. The electric power management with 
PHEVs are examined in 11211 . 
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Fig. 3. Average QoSEs of three residents Fig. 4. Energy levels of three Li-ion batteries Fig. 5. QoSEs for three residents with dif- 
(V = Vmax). (V = Vmax). ferent service contracts (V = Vmax/2). 
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Fig. 6. MG operation traces of the proposed algorithm for the 5-day period. Fig. 7. MG operation traces of proposed algorithm for the 7-day period. 



Microgrid is a new grid structure to group DRERs and 
local residents loads, which provides a promising way for the 
future SG. In [4], the authors review the MG structure with 
distributed energy resources. In [|22l . the integration of random 
wind power generation into grids for cost effective operation 
is investigated. In |23|, the authors propose a useful online 
method to discover all available DRERs within the islanded 
mode mircogrid and compute a DRER access strategy. The 
problem of optimal residential demand management is studied 
in II24II . aiming to adapt to time- varying energy generation and 
prices, and maximize user benefit. In [25], the authors investi- 
gate energy storage management with a dynamic programming 
approach. The size of the ESS's for MG energy storage is 
explored in ll26l . 

Lyapunov optimization is a useful stochastic optimization 
method 0. It integrates the Lyapunov stability concept of 
control theory with optimization and provides an efficient 
framework for solving schedule and control problems. It has 



been widely used and extended in the communications and 
networking areas (SI, [[TTI . In two recent work [|27l . [|28l . the 
Lyapunov optimization method is applied to jointly optimize 
power procurement and dynamic pricing. In [27 1, the authors 
investigate the problem of profit maximization for delay tol- 
erant consumers. In [|28l , the authors study electricity storage 
management for data centers, aiming to meet the workload 
requirement. Both of the work are designed based on a single 
energy consumption entity model. 

VI. Conclusion 

In this paper, we developed an online adaptive electricity 
scheduling algorithm for smart energy management in MGs 
by jointly considering renewable energy penertration, ESS 
management, residential demand management, and utility mar- 
ket participation. We introduced a QoSE model by taking 
into account minimization of the MG operation cost, while 
maintaining the outage probabilities of resident quality usage. 
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We transformed the QoSE control problem and ESS manage- 
ment problem into queue stability problems by introducing the 
QoSE virtual queues and battery virtual queues. The Lyapunov 
optimization method was applied to solve the problem with 
an efficient online electricity scheduling algorithm, which 
has deterministic performance bounds. Our simulation study 
validated the superior performance of the proposed approach. 
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Appendix A 
Derivation of Equation (fTsl) 

With the drift defined as in ([Tt]) , we have 



A{e{t)) = lE\j2m{t+i)r-{Xk{mxk{t)]+ 
ifc=i 

N ^ 
J2{iZn{t + l)f - {Z^{t)f\Z„{t)]\ 
n=l J 
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Where B = ^ Ef=i (max{I)--, ^m)' + iEn=i(2 + 
is a constant. 

Appendix B 
Proof of Lemma[I] 

Proof: In part 1) of Lemma [T] if Q{t) > 0, we have 
S{t) = according to (|7]). The the objective function of 
problem (|2T1) becomes 
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Y,{Zn{t)^an{t))pn{t). (26) 

n=l 

We first prove Lemma [Mil). If Xk(t) > -VC{t), we 
assume Rk{t) > 0. Then we have Dk{t) = according to ©. 
Accordingly, the object function (l26l) is transformed to 
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J2Mt)iRiit) - Di{t)) - (Z„(t) +a„(t))p„(t). 
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The above inequality is due to Xk{t) > —VC{t) and Rk{t) = 
Pit) + Q{t) - E,^fe(^i W - Diit)) - Ell PnW > 0. The 
last expression shows, given the assumption Rk{t) > 0, we 
may find another feasible electricity allocation scheme Q{t) = 



^i^k (^^ (^) ~ (^) ) + Z)n=i (^) ~ ^(^) ' which can achieve 
a smaller objective value by choosing Rk{t) = and Dk{t) = 
0. This contradicts with the assumption Rk{t) > 0. Thus, we 
prove that Rk{t) = when Xk{t) > -VC{t), under the 
situation Q{t) > 0,S{t) = 0. 

We then prove the second part of Lemma [TlfTal). It fol- 
lows © that Rk{t) = if Dk{t) > 0. Then (|26]) becomes 

VQ{t)C{t) + ^X,(t)(i?,(t) - AW) - 
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The above inequality is due to Xk{t) < —VC{t) < 
and AW = -^W - QW + E^^kiMt) + Dk{t)) + 
Yln=iPn{'^) > 0- The last expression shows, given the 
assumption Dk{t) > 0, we may find another electricity 
allocation scheme with Q{t) = ^i^k{Ri{t) — Diit)) + 
Yln=iPn{i) — P{t)^ which can achieve a smaller objective 
value by choosing Rk{t) = and Dk{t) = 0. This contradicts 
with the assumption Dk{t) > 0. We thus prove that Dk{t) = 
when Xk{t) < -VC{t), under the situation Q{t) > 0, S{t) = 
0, which completes the proof of Lemma [TlfTal). 

We next prove Lemma [T][Tbl). For the first part, if Zn{t) > 
VC{t) — an{t), we assume < Pn{t) < (1 — Sn)an{t)- 
Following ([2Q1) and S{t) = 0, we have 
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(1 - 8n)an{t)]C{t) ^Y.Xk{t){Rk{t) - Dk{t)) + 

Zn{t){l - Sn)an{t) - {Zn{t) + an{t)){l - Sn)an{t). 

The above inequality is due to Zn{t) > VC{t) — an{t) and the 
assumption Pn{t) < (1 — Sn)(^n{t). The last equality shows, 
given the assumption Pn{t) < (1 — Sn)(^n{t), we may find an- 
other electricity allocation scheme with Pn{t) = {l — Sn)an{t) 
and Q{t) = EtiiMt) - Dk{t)) + Ej^uPji*) " m + 
(1 — Sn)an{t), which can achieve a smaller objective value. 
This contradicts with the previous assumption. Thus, we have 

Pn{t) > {l-6n)an{t). 

For the second part of Lemma [TIfTbl), assume Pn{t) > for 
< Zn{t) < VC{t) - an{t).lt follows © that S{t) = 0. The 
objective function (l2Ql) can be written as 
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B + VQ{t)C{t) ^^Xk{t)iRk{t) - D^{t)) + 
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> 5 + VQ{t)C(t) + W){Rk{t) - Du{t)) + 
Y^iZjim - 5j)aj{t) - (Zjit) + a^{t))pj{t)) + 
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> B^V[-P{t)^Y.^Rk{t)-Dk{t))^ 
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Y{Z,{t){l - 5^)a,{t)) - YiiZjit) + a,{t))pj{t)). 

The first inequality is due to < Zn{t) < VC{t) - anif) 
and the assumption Pn{t) > 0. The second inequality is 
due to the non-negativity of Zn{t) and an{t). The last 
equation shows, given the assumption Pn{t) > 0, we may 
find another electricity allocation scheme with Pn{t) = and 
Q{t) = -P{t) + Zk=,{Rk{t)-Dk{t)) + Zj^^Pj{t), which 
can achieve a smaller objective value. This contradicts with 
the previous assumption. Thus, we have Pn{t) = 0, which 
completes the proof of Lemma fllTbl). 



In part El) of Lemma [B if S{t) > 0, we have Q{t) = 
according to (|7]). The objective function (|2T1) becomes 

K 

-VS{t)W{t) ^YXk{t){Rk{t) - Dk{t)) - 
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Y,{Zn{t)^an{t))pn{t). (27) 
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We can prove part O with a similar approach as in the case 
of part [T] The detailed proof is omitted for brevity. ■ 

Appendix C 
Proof of Lemma[2] 

Proof: Since < Cmin < C{t) < Cmax and V > 0, 
we have Rk{t) = when Xk{t) < —VCmax, and Dk{t) = 
when Xk{t) > —VCmin according to Lemma [T][T]). Similarly, 
since < Wmin < W{t) < Wmax and F > 0, we obtain 
Rk{t) = when Xk{t) < -VWmax. and Dk{t) = 
whenXfe(t) > —VWmin according to Lemma [TJO 

Since Cmax > Wmax and Cmin > Wmin, wc concludc 
that if Xk{t) > —VWmin, the optimal solution always select 
Rk{t) = 0. If Xk{t) < —VCmax, the optimal solution always 
select Dk{t) = 0. The proof is completed. ■ 
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